Linguistically-aware attention for reducing the semantic gap in vision-language tasks

Attention models are widely used in Vision-language (V-L) tasks to perform the visual-textual correlation. Humans such a correlation with strong linguistic understanding of visual world. However, even best performing attention model V-L lacks high-level understanding, thus creating semantic gap between modalities. In this paper, we propose an mechanism - Linguistically-aware (LAT) that leverages object attributes obtained from generic detectors along pre-trained language reduce gap. LAT represents and textual modalities common linguistically-rich space, providing awareness process. We apply demonstrate effectiveness three tasks: Counting-VQA, VQA, Image captioning. novel counting-specific VQA predict intuitive count achieve state-of-the-art results on five datasets. Captioning, show nature by adapting it into various baselines consistently improving their performance.